Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning

Humans tend to decompose a sentence into different parts like sth do at someplace and then fill each part with certain content. Inspired by this, we follow the principle of modular design propose novel image captioner: learning Collocate Visual-Linguistic Neural Modules (CVLNM). Unlike widely used neural module networks in VQA, where language (i.e., question) is fully observable, task collocating visual-linguistic modules more challenging. This because only partially for which need dynamically collocate during process captioning. To sum up, make following technical contributions train our CVLNM: (1) distinguishable design—four encoder including one linguistic function words three visual content noun, adjective, verb) another decoder commonsense reasoning, (2) self-attention based controller robustifying (3) part-of-speech syntax loss imposed on further regularizing training CVLNM. Extensive experiments MS-COCO dataset show that CVLNM effective, e.g., achieving new state-of-the-art 129.5 CIDEr-D, robust, being less likely overfit bias suffering when fewer samples are available. Codes available https://github.com/GCYZSL/CVLMN.